DOCUMENT RESUME 

ED 342 437 JC 920 119 



AUTHOR 
TITLE 
PUB DATE 
NOTE 

PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



Armstrong, Judy 

Assessing Skills froia Placement to Completion. 
15 Nov 91 

46p.; Paper presented at the North Texas Community 
College Conference (November 15, 1991). 
Speeches/Conference Papers (150) — Guides - 
Non-Classroom Use (055) 

MF01/PCC2 Plus Postage. 

*Achievement Tests; ^College Outcomes Assessment; 
Graphs; Higher Education; *Outcomes of Education; 
Remedial Instruction; Scores; *Student Evaluation; 
*Student Placement; "Testing Programs; Two Year 
Colleges; Universities 



ABSTRACT 

A system to provide objective measures of 
institutional effectiveness was implemented at the Roswell branch of 
Eastern New Mexico University to determine whether the college was 
accountable to students, staff, and taxpayers; to improve the 
curriculum and programs; and to prepare for accreditation review in 
1931. A task force spent the summer of 1988 researching and reviewing 
various forms of outcomes assessment. The program now consists of the 
following: entering students take placement tests in reading, 
English, and mathematics using the "Asset"; students in "College 
Success," a required course for entering students are given the 
Critical Thinking test; and after the completion of 45 units, the 
tests are retaken to obtain a post-test score. In addition, the 
school began tracking developmental studies students through their 
college coursewcrk. Results of the testing efforts included the 
following: (1) in 1989-90, 46% of the entering students passed the 
math placement test, 49% passed the English placement test, and 59% 
passed the reading placement test; (2) in 1990-91, when Roswell 
switched to the "Asset" placement test, only 34% passed the math 
section, 32% passed the English test, and 35% passed the reading 
test; and (3) in 1989-90, 96% of the students enrolled in 
developmental math classes passed, 82% passed developmental English 
classes, and 83% passed developmental reading, while in 1990-91, pass 
rates were 88% for math, 73% for English, and 84% for reading. One of 
the advantages of the Roswell' s testing program is that it 
facilitates comparisons with similar institutions and national norms. 
(JMC) 
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ASSESSING SKILLS FROM PLACEMENT TO COMPLETION 
JUDY ARMSTRONG 
NOVEMBER 15, 1991 

(transparency, Great Year) I am very pleased to be here today Perhaps the first 
thing I should do is introduce myself and tell you a little bit about my school. 
My name is Judy Armstrong, and I am Assistant Dean of Instructional Support a% 
the Roswell branch of Eastern New Mexico University, a position I've held for the 
past three years. In this position, I oversee a variety of programs. One of my 
primary responsibilities is to serve as division chair for the Division of 
Developmental Studies. I also supervise the tutoring program, the College 
Success and Critical Thinking classes, the Adult Basic Education Program, all of 
the university's testing programs (GED, placement test, NTE, etc.), the Learning 
Resource Center, services for the Deaf and Disabled, the Single Parent/Displaced 
Homemaker Program, the Carl Perkins Grant, and the Vocational Training Program 
for the Developmentally Disabled. As you can see, whenever they don't know 
exactly where a new program fits, they give it to me. So it was only natural 
that when the university began its consideration of an outcomes assessment 
program that I should be the one blessed with implementing it. 

A branch of Eastern New Mexico University, which is located in Portales 90 miles 
northeast of Roswell, ENMU-R began in 1957, with 157 part-time students who met 
for evening classes at the high school. When a new post office was opened in 
1963, the old Federal Building became the college's new home, Roswell' s first 
daytime, coeducational college with 8 classes serving 75 students. The closing 
of Walker Air Force Base in 1967 created a local crisis, but presented an 
opportunity for the college, which secured 27 brick buildings and numerous frame 
buildings situated on 234.5 acres near the center of the former military base, 
and was able to expand its offerings considerably. In 1982, the voters of our 
district approved a five-million-dollar bond issue, which was combined with state 
appropriations to finance a building program which resulted in a beautiful new 
campus, which was completed in 1989. 



Funding for the school is formula driven, based on full-time enrollment. With 
the oil boom of the early 1980' s, the campus was riding high, primarily because 
of our Oilfield Training Center, one of the few programs of its kind in the 
nation, which attracted hundreds of full-time students from all over the world. 
However, the oil decline brought a crisis for the school as well. Enrollment in 
the OTC dropped to as few as 6 students. Faced with the possibility of having to 
RIF faculty and staff, the school formed a task force to create an aggressive 
marketing, recruiting, and retention plan. I am happy to say that plan worked, 
for this semester we had an all-time high enrollment of 2,074 students, with an 
FTE of 1,226. Even better, that enrollment was spread throughout the university 
curriculum, including a large number of academic transfer students as well as 
many vocational -technical, two-year degree seeking students. 
While we are a branch of a university, we function as a community college. 
Although we share many policies and some curriculum with our mother campus, we 
also have developed our own internal systems, one of them is outcome measurement. 
Please feel free to ask questions at any time during my presentation. 
1. Motivation: 

A. Several things have motivated us to implement a system to objectively measure 
our institutional effectiveness. All of us have read the national reports 
indicating poor student performance. Almost every day another headline is 
screaming at us about how poorly the nation's schools are doing, 
(transparencies) Our country's two-year institutions have become the "hospitals" 
if you will for those students whose skills, both personal and academic, are 
lacking. We have been charged with the task of curing those students and 
bringing their skills, which they haven't learned in the past 18, 30, or even 50 
years, up to snuff in one semester. We have willingly accepted this mission and 
have done our best to control the epidemic. But have we really done what we set 



out to do? Or have we just given our students placebos and sent them on their 
way to failure in the real world? ENMU-R wanted to know if it was truly being 
accountable to its students, to their employers, and to the taxpayers. 

B. We also entered into an assessment program to help us improve our curriculum 
and programs. If our students are indeed adding value to their education by 
attending ENMU-R, that's great. But if they are not making gains during their 
stay at our school, why aren't they? And what can we do to change our programs, 
curriculum, support services, or method of instruction to help them to succeed? 

C. (motivation transparency) Of course, one of the big motivating forces was the 
fact that our institution was to be evaluated for continuing accreditation in 
1991, and North Central had just required all of its accredited institutions to 
have an assessment program in place. Now, I don't want to imply that North 
Central was standing over us with a whip, but their impending visit certainly did 
speed up the process. As a matter of fact, our dean of the college, Dr. Denis 
Roark, had been the chairman of the NCA self- study effort during our previous 
evaluation, and it is because of his foresight that we already had a plan in 
existence and working when the evaluators came this fall and didn' t have to try 
to design and implement an assessment program as well as do an institutional 
self-study. By the way, as you might guess, I was the coordinator of the 
accreditation self- study this year, and I'm pleased to tell you that we got our 
first-ever 10-year accreditation! Besides North Central, the New Mexico 
Commission on Higher Education and the New Mexico State Department of Education 
were also becoming more and more adamant that the state's schools adopt some 
system of accountability. 

2. background. So, a task force made up of the vocational and academic division 
chairs, faculty and student representatives, and the Dean of the College spent 
the summer of 1988 researching and reviewing various forms of outcome assessment 



being done. Our first decision as a task force was to utilize a standardized 
testing instrument to determine a value added measurement of our students* during 
their first two years at ENMU-R. This would be examined along with other outcome 
measures- -number of graduates, transfers to 4-year institutions, placement, 
student evaluations and follow-up surveys, evaluations by outside agencies, on- 
going program review, licensing results for example- -to determine institutional 
and program effectiveness. Once the decision had been made to utilize 
standardized tests, the committee researched and evaluated various instruments on 
the market. Because we did not want to test our students more than necessary, we 
decided to use the ACT CAAP (Collegiate Assessment of Academic Proficiency) as 
our instrument because it can be correlated with the ACT Asset Placement test 
designed for two-year institutions. Therefore, by adding only one 50-minute test 
to the placement testing system that was already in place, we would have baseline 
scores for virtually all of our academic and vocational students in English, 
reading, math, and critical thinking. Also, since cost is always a factor, by 
changing to the Asset placement test, we could initiate our outcome assessment at 
a minimum expense, rather than testing our students twice on the more costly 
CAAP. I must also say that after previewing the CAAP, the task force felt that 
it was too advanced to give to entering freshmen- -at least ours. If they were 
faced with a test at that level of difficulty during the first few weeks of 
school, they might get so overwhelmed that they would drop out and we wouldn't 
have anyone to post- test! Another factor affecting our decision to use these 
instrument is that the main campus had already begun outcome assessment and was 
using ACT's COMP (College Outcome Measures Program). Although we felt that the 
COMP was not the appropriate instrument for our purposes, we hope that since they 
are both ACT instruments some correlation exists. Finally, our provost Dr. Loyd 
Hughes is concerned with how we are doing as an institution compared to other 2- 
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year institutions across the country. Therefore, all of our assessment and 
evaluation instruments, including the faculty student evaluations, are nationally 
normed, as are the Asset and the CAAP. 

3. (Assessment flow chart) So here is the plan we came up with. As you can 
see, it includes a 'variety of testing opportunities. First, all entering 
students are tested on the Asset, and the CAAP Critical Thinking module is 
administered to almost all freshmen through our College Success classes, which 
are required for virtually all students during their first semester. Students 
who have completed 45 credit hours of work within a three-year time-span will 
post-test on the CAAP reading, writing, mathematics, and critical thinking 
modules. Vocational students will post- test on the SOCAT, which I will explain 
later, and nursing and aviation maintenance students will post test on their 
individual licensing exams. The results, complete with a value added score for 
academic students, will be shared with the Instructional Council, which is made 
up of all of the division chairs, and other appropriate university individuals 
and groups for review. 

4. (key points transparency #1) As I said, all entering students are required to 
take the placement test, with the following exceptions. Students who do not 
score at required university skills levels must take the appropriate 
developmental classes to achieve the required levels of proficiency in English, 
math, and reading. 

5. (1989-90 pt results) These results of our placement testing are based on our 
1989-90 testing period since we did not begin using the Asset until after summer 
registration, 1990 (scores based on College Board Assessment and Placement Test 
for Community College Students). 

(1990-91 placement test results) These results of placement testing are 
based on our first year of Asset use. One thing I particularly like about the 
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Asset is that it provides for computer scanning and reporting of the test 
information. It also provides the ability to gather information that is useful 
to the institution. For instance, we are using the local option questions to 
assist our marketing program, asking such questions as; "How did you hear about 
ENMU-Roswell?" and "Which radio/tv station do you listen to most frequently." 
Having this information will help us to focus our publicity on targeted groups. 
Also, the Asset allows us to generate reports and information that we have not 
been able to gather in the past because of time restraints. For example, I can 
now tell how students from each school in our service district performed on the 
tests, information which we share with the schools. From this information we 
have discovered what we suspected but could not substantiate, that students from 
our local alternative high school, the small rural high schools in the valley, 
the Job Corps, and GED graduates score considerably lower on the placement test 
than do high school graduates from the larger schools in our district. We also 
receive from ACT an Entering Student Descriptive Report, which details the 
characteristics of an entire group of assessed students each semester. 

6. (Developmental Studies follow up transparency) Actually, we began our 
program of accountability when we changed the policy regarding our developmental 
classes several years ago. Students enrolled in these classes must take an 
alternate form of the placement test as their final exam. They must pass the 
exam in order to pass the class. We felt chat class grades, because they allowed 
for retakes of tests, did not adequately indicate whether or not students had a 
firm grip on the skills they were originally tested on. If students do not pass 
the post -test, they must repeat the class. As you can see, the classes have had 
a fairly high level of success, for a large majority of students pass the post- 
test after just one semester. 

7, (DL> tracking results transparency) In addition, we have begun tracking our 



developmental studies students through their college coursework. We also receive 
from ACT a Returning Student Retention Report, a detailed report that provides a 
look at student outcomes, including success in the classroom and enrollment and 
re-enrollment patterns. Beginning last semester, we flagged our graduates in our 
computerized student data base, so we will now be able to identify graduates who 
have coma through the developmental studies program and do some comparisons 
between them and students who do not require remedial instruction. This tracking 
of developmental studies students will take place on an annual basis. 
8. (key points of assessment transparency #2) As you can see, we have designed 
our assessment program to be as non-intrusive as possible. Since students were 
already required to take the placement test and the critical thinking module of 
the CAAP is given during a class period of College Success, no additional student 
time is required for initial testing. CAAP, the Collegiate Assessment of 
Academic Proficiency, consists of b objective tests- -reading, writing, math, 
critical thinking, and science reasoning. A school may use any combination of 
modules. CAAP also can provide a writing sample which can be scored locally or 
by ACT. Although the task force strongly recommended a writing sample as part \£ 
the assessment program, costs have prohibited our using one at this time; 
however, it will more than likely be added sometime in the future. Our school 
chose to use just four modules for post- testing, since we felt the science 
reasoning test was very much like a reading test. ACT scores the tests very 
quickly, and sends us a summary of the data, plus two copies of the students' 
score report for each test taken. One copy is given to the student and the other 
copy is kept in the student's permanent record in the Admissions Office, after it 
has been logged into the computer data bank. The summary provides descriptive 
statistics and frequency distributions for the institution's total group and 
general comparative information. The math, writing, and reading modules also 
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provide subscores to help institutions pinpoint problem areas. 

9. (Critical Thinking results) This shows you how our students have done on the 
critical thinking module so far. Since we have not yet completed our first round 
of post- testing, we do not have comparative dat-a for that yet. As you can see, 
our students are a little above the norms for 2-year college students and a 
little below national norms for all freshmen. 

10. (key points of assessment transparency #3) The tests, as I said before, are 
non- intrusive. The 50-minute module fits in well with class periods for College 
Success. Post - testing will take approximately 4 hours, and we will have a 
morning, afternoon, evening, and Saturday testing periods so that all identified 
students will have the opportunity to participate. Students having completed 45 
hours within a 3 -year timeframe will be identified; contacted personally, and 
asked to commit to a testing period. Students will be excused from classes to 
test, but classes will not have to be dismissed. Student government will assist 
in gathering incentives for students to take the post-tests, and we have 
published a statement of students' required participation in the catalog. Post- 
test scores, like the baseline scores, will be distributed to students. 
Obviously, this will require a great "marketing" effort by our faculty, staff, 
and student government to assuv - adequate participation and optimum effort on the 
students' part. However, our initial student reaction to the pre- testing, as 
seen through the administration of the critical thinking test in College Success 
classes, and post- testing, through our SOCAT exams, has been extremely positive. 
The students seem interested in measuring themselves and in helping the school to 
improve as much as possible. Of course, whether that interest translates into 
great hordes of students showing up to take the post- test or not remains to be 
seen this coming semester when we do our first round of academic post- testing. 

11. (ACT profile transparency, trends) We have also received a profile of 
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students who took the ACT and requested scores to be sent to our institution. 
This is helpful additional information in reviewing our total student outcome, as 
is the retention report, graduation rates, and all of the other outcome measures 
that we study. 

At the completion of their programs, vocational students are post -tested in two 
says. (licensing results transparency) Most of *:he graduates of the nursing, 
EMS, and aviation maintenance technology programs take licensing exams, and these 
are the results from 1989-90. I do not have all of the results from last year's 
graduates available yet, but I did just hear that 91% of last year's nursing 
graduates passed the licensing exam (29 passed out of 32 taking; one did not 
take) . 

(SOCAT results transparency) Certificate and degree students in all other 
vocational programs, with the exception of Petroleum Technology and Industrial 
Technology for which no standardized test exists, take the Student Occupational 
Competency Achievement Test (SOCAT) during their final semester. These tests, 
put out by NOCTI (National Occupational Competency Assessment Institute) also 
provide national norms. The tests consist of a written exam, which covers 
factual knowledge, technical information, understanding of principals, and 
problem solving, and a performance exam, which consists of work assignments 
designed to sample the manipulative skills required in the occupation. The SOCAT 
also gives the institution subtest scores which indicate the students 1 
performance on specific subject areas, such as brakes. 

12. After all of this testing, how does the institution utilize the abundance of 
information it receives? SOCAT results are given directly to the division chair 
and instructors in each of the programs as soon as they are received. Thus, 
information can be utilised immediately to improve curriculum if necessary for 
the next group to be tested. This information, as well as the academic outcomes 
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information, will be shared on an annual basis with the deans, division chairs, 
the Instructional Council, and the Administrative Council to utilize as they work 
on program review, new programs, additions to the curriculum, and overall 
institutional evaluation. 

13. (advantages of assessment program transparency) One way to measure an 
institution's effectiveness is by comparing its outcomes with other similar 
institutions. All of these standardized instruments allow for comparison with 
national norms, (b) Students receive individual results, which they can use in 
their job searches and on their resumes. It also gives them a sense of ownership 
in the program and shows them how they stack up against other students, (c) 
Testing is non- intrusive and does not disrupt regular class schedules, (d) Data 
is scored and results generated by outside agencies, which provides students, and 
the institution, with a validation from outside sources of their competencies, 
(e). The institution receives support from testing professionals. This is 
especially important in our school, where we do not have a research officer or 
other person identified to input and analyze data. The reports generated by ACT 
and the SOCAT are easily transferred into institutional reports to share with the 
university community and other agencies. Also, and this is extremely important, 
the people from both of these agencies are very helpful. Whenever I have a 
question, I 'ust pick up the phone and call Roy Henderson at ACT, and if he 
doesn't know the answer, he at least points me to someone who does. Also, ACT 
sponsors a variety of workshops and seminars to discuss these very ideas and 
others, which lets people like us meet to learn more about assessment and other 
current issues, to express our own concerns, and to exchange information, ideas, 
and techniques. 

14. Our assessment program is well off the ground, but it is still in its 
infancy. Once we have completed our first round of post- testing, we will have a 
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much better feel for its strengths and weaknesses. We have recently added to our 
total outcome assessment plan placement information and employer satisfaction 
information. We all believe that our institutions and programs are doing a good 
job, and I would like to say thai: assessment is just a passing fad; however, that 
is not the case (St. Peter transparency). I'm afraid it's with us to the end, 
and that's good. The technology of the next century is going to demand higher 
and higher skills of our students, and we must be sure that they possess those 
skills when they leave our doors. Student success is what we're all after, and I 
would like to close with this National Student's Prayer, by Philip M. Summers, 
President of Vincennes University: 

15. Thank you for your attention. If you have any questions, I would be happy 
to address them at this time. 



11 

14 



NATIONAL STUDENT'S PRAYER 



Philip M. Summers 
President 

Vincennes University 

Heavenly Father, look down with favor on the humble students who travel 
the campuses of this nation asking their parents for money, carrying 
armloads of occasionally-used textbooks, and seeking in-depth educational 
inspiration and insight at late-night dances. 

We pray that they find housing accommodations that will include space for 
their stereos and televisions and other necessities of life, that their 
cars are not too much newer or more costly than the faculty's, and that 
they understand that they should not write checks for amounts greater 
than they have in their bank accounts* 

Lead them to good study habits, and may they learn that daytime is for 
activity and nighttime is for sleep and not the reverse. 

Give them strength to attend class regularly and be prepared for tests, 
and may they be creative with their reasons when this cannot be done. 

Protect the female students from pleas of eternal love from male 
students. Lead them not into temptation, for they know not what they do. 

Save the male students from making pleas of eternal love which come not 
from the heart, but from other parts of the anatomy; and please do not 
forgive them their trespasses, for they know exactly what they do . 

And when they graduate, grant them the favor of finding employment or 
transfer to another college, and may they remember their Alma Mater as a 
great place that helped them in their lifes. 

This we ask in the name of students who like the cafeteria food, 
commuters who can find parking spaces, and athletes who do go to class, 
and all the dedicated, hard-working, and understanding deans, faculty and 
professional staff who continue to go the extra mile to help students 
achieve success. 
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Key Points of Assessment Program 

All entering students ire requited to tike placement 
tests with the following exceptions: 

1. Students who have ilready earned a minimum of 39 hours of college 
credit, which must include English and mathematics, 

2. Students with fewer than 3D hours of college credit who transfer with 
a passing grade in freshman level courses In English and mathematics are 
exempt from those portions of the test, but are still required to take the 

* reading portion of the test 

3. Part-time, non-degree students unless the) plan to enroll in English 
and/or mathematics. 
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Eastern New Mexico University - Roswell 

1990-1991 Asset Placement Test Results 
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Eastern New Mexico University-Roswell 

1989-90 Developmental Studies Class Results 
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Eastern New Mexico University - Roswell 

1990-1991 Developmental Studies Class Results 
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Eastern New Mexico University - Roswell 

1990 Developmental Studies Followup Results 




GPA Avg. 9.34 87 



GPA Avwiq* For All 1990 GradutUt - 3.27 



Key Points of Assessment Program 



Students are given placement tests in reading, English 
and mathematics using ASSET. 

Students in College Success (required course for 
most entering students) are given the Critical Thinking 
module of the CAAP. 

The ASSET and CAAP Critical Thinking scores are used 
as baseline measurements. These students are tracked 
s After two years, academic students from this group who 
have a minimum of 45 credit hours are required to take the 
CAAP reading, English, mathematics, and critical thinking 
modules to obtain a posHest score. 
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Eastern New Mexico University - Roswell 

GAAP Critical Thinking Test 
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Key Points of Assessment Program 



Scores from baseline and post-test are correlated to obtain 
a value-added score. Both ASSET and GAAP are ACT 
Instruments which can be correlated. 



Both btse/ine and post-test ere designed to be as non-intrusive 
as possible, Students can select time to be tested. Norming 
results are available for both Instruments. 



Vocational students are tested via SOCAT to obtain competencies 
as compared to other students, state and national norms. 

Results of tests provide information for program strengths and 
weaknesses. Instructional Council provides the vehicle for 
modifying program io address weaknesses. 
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English 


2.48 


2.71 


2.61 


3.02 


2.74 


Mathematlca 


Z62 


2.06 


2.47 


2.76 


2.46 


Social Studies 


2,61 


2.63 


2.86 


3.12 


2.61 


Natural Sciences 


2.63 


Z60 


2.61 


2.92 


2.62 


Av«rse» of QnidM (H8A) 


2.61 


2.60 


2.68 


2.95 


Z66 
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Eastern New Mexico University-Roswell 

1989-90 State/Federal Licensing Exam Results 



Passed Failed 

warn 

% of Students Who Passed or Failed 



Not Tested 




FAA 

Gen 



FAA 
Airfr 



FAA Nursing 
PP RN 

Areas Tested 



Nursing 
LPN 



EMS 
Basic 



EMS 
Inter 



Eastern New Mexico University - Roswell 

1989-91 SOCAT Results 



ENMU-R National ENMU-R National 
Writtan Written Parform. Parform. 




AooVBk Auto Moh Comp Prog Etootronloa GanOffloa WaMIng 



~ v 1* 
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Advantages of Assessment Program 



Standardized instruments provide 
comparisons to other institutions. 

Students receive individual results. 

Testing is designed to be non-intrusive 
in terms of time and scheduling. 

. Data is scored and results generated 
by outside agency. 

Institution receives support from testing 
professionals. 



ERIC Clearinghouse for 
Junior Colleges 



APR 1« 1992 



